{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"2.2.2-Query Auto Completion Hands-On Session.ipynb","provenance":[]},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"lJa_fge2V6Tu","colab_type":"text"},"source":["# Introduction\n","Query auto completion (QAC) is an important component of writing assitance in search engines. Given an incomplete user input, QAC suggests a list of possible completions that help save users' keystrokes. These completions usually start with the user input. For example, when users type in \"linke\", QAC may suggests \"linkedin\", \"linkedin corporation\", etc..\n","\n","A typical QAC system usually consists of two main components: candidate generation and candidate ranking. Given a user input, the candidate generation component generates a list of candidate suggestions first, and the candidate ranking component ranks the suggestions in the order of relevance. \n","\n","In this session, we will go through the steps of building a QAC system. The content covers\n","\n","\n","1.   sample dataset preparation\n","2.   candidate generation component construction\n","3.   candidate ranking component construction\n","4.   a QAC system in action\n","\n"]},{"cell_type":"markdown","metadata":{"id":"m5APesp5anLv","colab_type":"text"},"source":["# Sample Dataset Preparation\n","In this section, we will use a small sample of a public dataset, AOL, to generate the queries.\n","\n","The outputs are:\n","1.  background queries (used for candidate generation)\n","2.  training / development / test queries (used for model training and evaluation in candidate ranking)\n","\n"]},{"cell_type":"code","metadata":{"id":"tZxfmXLqTads","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":204},"executionInfo":{"status":"ok","timestamp":1598312795687,"user_tz":420,"elapsed":4353,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}},"outputId":"9e0cc307-4f45-4e8c-d656-8000aa153f51"},"source":["# Download a sample data of the AOL (America Online) dataset from the web\n","# The complete collection consists of ~20M web queries collected from ~650k AOL users over three months (2006 March to 2006 June).\n","!wget http://www.cim.mcgill.ca/~dudek/206/Logs/AOL-user-ct-collection/user-ct-test-collection-01.txt"],"execution_count":1,"outputs":[{"output_type":"stream","text":["--2020-08-24 23:46:31--  http://www.cim.mcgill.ca/~dudek/206/Logs/AOL-user-ct-collection/user-ct-test-collection-01.txt\n","Resolving www.cim.mcgill.ca (www.cim.mcgill.ca)... 132.206.73.26\n","Connecting to www.cim.mcgill.ca (www.cim.mcgill.ca)|132.206.73.26|:80... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 222762553 (212M) [text/plain]\n","Saving to: ‘user-ct-test-collection-01.txt.1’\n","\n","user-ct-test-collec 100%[===================>] 212.44M  62.1MB/s    in 3.7s    \n","\n","2020-08-24 23:46:35 (57.9 MB/s) - ‘user-ct-test-collection-01.txt.1’ saved [222762553/222762553]\n","\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"JSiDQNLCVEPO","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":187},"executionInfo":{"status":"ok","timestamp":1598312795688,"user_tz":420,"elapsed":4343,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}},"outputId":"3467941c-a0f8-4165-9be8-7cb849023097"},"source":["# Take a glimpse into the data. We will only make use of the \"Query\" and the \"QueryTime\" column\n","!head user-ct-test-collection-01.txt"],"execution_count":2,"outputs":[{"output_type":"stream","text":["AnonID\tQuery\tQueryTime\tItemRank\tClickURL\n","142\trentdirect.com\t2006-03-01 07:17:12\t\t\n","142\twww.prescriptionfortime.com\t2006-03-12 12:31:06\t\t\n","142\tstaple.com\t2006-03-17 21:19:29\t\t\n","142\tstaple.com\t2006-03-17 21:19:45\t\t\n","142\twww.newyorklawyersite.com\t2006-03-18 08:02:58\t\t\n","142\twww.newyorklawyersite.com\t2006-03-18 08:03:09\t\t\n","142\twestchester.gov\t2006-03-20 03:55:57\t1\thttp://www.westchestergov.com\n","142\tspace.comhttp\t2006-03-24 20:51:24\t\t\n","142\tdfdf\t2006-03-24 22:23:07\t\t\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"fgJUs756XCPS","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1598312795919,"user_tz":420,"elapsed":4565,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}},"outputId":"4335e6a0-f7fc-4be4-c767-c6bf80246715"},"source":["!wc -l user-ct-test-collection-01.txt"],"execution_count":3,"outputs":[{"output_type":"stream","text":["3558412 user-ct-test-collection-01.txt\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"dMh53nviXGnP","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":102},"executionInfo":{"status":"ok","timestamp":1598312797937,"user_tz":420,"elapsed":6573,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}},"outputId":"e1978a1d-905f-45f8-cf19-4316faba6911"},"source":["# Extract query and split background/train/dev/test data based on QueryTime\n","import string, re, random\n","random.seed(1234)  # control repeatability\n","\n","sample_ratio = 0.01\n","\n","punctuation = string.punctuation\n","translator = re.compile('[{}]+'.format(re.escape(string.punctuation)))  # regex to identify consecutive punctuations\n","\n","with open('user-ct-test-collection-01.txt', 'r') as fin:\n","  bkg_queries = list()\n","  train_queries = list()\n","  dev_queries = list()\n","  test_queries = list()\n","\n","  for i, line in enumerate(fin):\n","    if i == 0 or random.uniform(0, 1) > sample_ratio:\n","      continue\n","    split_info = line.split('\\t')\n","    \n","    q = translator.sub(\" \", split_info[1].lower().strip()) # replace punctuation with space\n","    t = split_info[2]\n","\n","    if \"2006-03-01 00:00:00\" <= t < \"2006-05-01 00:00:00\":\n","      bkg_queries.append(q)\n","    elif t < \"2006-05-14 00:00:00\":\n","      train_queries.append(q)\n","    elif t < \"2006-05-21 00:00:00\":\n","      dev_queries.append(q)\n","    elif t < \"2006-05-28 00:00:00\":\n","      test_queries.append(q)\n","    else:\n","      pass\n","\n","print(\"Number of produced queries:\")\n","print(\"  background queries\\t{}\".format(len(bkg_queries)))\n","print(\"  train queries\\t{}\".format(len(train_queries)))\n","print(\"  dev queries\\t{}\".format(len(dev_queries)))\n","print(\"  test queries\\t{}\".format(len(test_queries)))"],"execution_count":4,"outputs":[{"output_type":"stream","text":["Number of produced queries:\n","  background queries\t23640\n","  train queries\t5057\n","  dev queries\t2431\n","  test queries\t2887\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"dqKfWG5zfBw3","colab_type":"text"},"source":["# Candidate Generation Component Construction\n","In this section, we will use the background queries to construct a finite state transducer (FST) based candidate generator. The process is similar to a trie look up.\n","\n","Given a query \"abc\", the FST will look up top k most frequent background queries that starts with \"abc\"."]},{"cell_type":"code","metadata":{"id":"turkqPDHfKA-","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":68},"executionInfo":{"status":"ok","timestamp":1598312800570,"user_tz":420,"elapsed":9194,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}},"outputId":"0c7b7e47-0204-432a-e777-9bf24454885b"},"source":["# Install an open-source autocompletion package that supports trie-based completion (https://zepworks.com/posts/you-autocomplete-me/)\n","# You can also use ElasticSearch for the same purpose (https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters.html#completion-suggester)\n","!pip install fast-autocomplete"],"execution_count":5,"outputs":[{"output_type":"stream","text":["Requirement already satisfied: fast-autocomplete in /usr/local/lib/python3.6/dist-packages (0.6.0)\n","Requirement already satisfied: python-Levenshtein>=0.12.0 in /usr/local/lib/python3.6/dist-packages (from fast-autocomplete) (0.12.0)\n","Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from python-Levenshtein>=0.12.0->fast-autocomplete) (49.2.0)\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"piNgzIvs94Tb","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1598312800834,"user_tz":420,"elapsed":9450,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}}},"source":["# Construct the query logs using background queries\n","# Write down the logs in a format accepted by the package\n","from collections import Counter\n","bkg_query_count = Counter(bkg_queries)\n","\n","query_log = dict()\n","for query, count in bkg_query_count.items():\n","  query_log[query] = [dict(), query, count]\n","\n","import json\n","with open('query_log.json', 'w') as outfile:\n","    json.dump(query_log, outfile, indent=2)"],"execution_count":6,"outputs":[]},{"cell_type":"code","metadata":{"id":"FG7nlbGyZpRg","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":187},"executionInfo":{"status":"ok","timestamp":1598312800835,"user_tz":420,"elapsed":9443,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}},"outputId":"c2bdf511-beab-4cac-8fde-246d94910c6b"},"source":["!head -n 10 query_log.json"],"execution_count":7,"outputs":[{"output_type":"stream","text":["{\n","  \"staple com\": [\n","    {},\n","    \"staple com\",\n","    1\n","  ],\n","  \"lottery\": [\n","    {},\n","    \"lottery\",\n","    2\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"mKOt6vCkXUzm","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1598312801792,"user_tz":420,"elapsed":10393,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}}},"source":["from fast_autocomplete import autocomplete_factory\n","\n","content_files = {\n","    'words': {\n","        'filepath': 'query_log.json',\n","        'compress': True  # means compress the graph data in memory\n","    }\n","}\n","\n","candidate_generator = autocomplete_factory(content_files=content_files)"],"execution_count":8,"outputs":[]},{"cell_type":"code","metadata":{"id":"sDEE4USDcEeo","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1598312801793,"user_tz":420,"elapsed":10387,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}},"outputId":"ac2c06e6-5254-42ae-aeef-57e34c2dcd39"},"source":["candidate_generator.search('goo', max_cost=0, size=5)"],"execution_count":9,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[['goo'], ['google'], ['google com'], ['googles'], ['google earth']]"]},"metadata":{"tags":[]},"execution_count":9}]},{"cell_type":"markdown","metadata":{"id":"mnd7B8ylfOlB","colab_type":"text"},"source":["# Candidate Ranking Component Construction\n","In this section, we will go through the steps to use the deep text understanding framework, [DeText](https://github.com/linkedin/detext), to build a CNN ranking model ([CLSM](https://www.microsoft.com/en-us/research/wp-content/uploads/2015/10/spir0468-mitra.pdf)) to rank the queries.\n","\n","This model uses CNN to extract semantic representations from the prefix and the suffix. We define prefix as the sequence of complete words in the user input, and suffix as the trailing words after the prefix in the completions.\n","\n","For example, for user input \"linkedin sof\" and candidate suggestion \"linkedin software engineer\", the prefix is \"linkedin\" and the suffix is \"software engineer\". \n","\n","After the representation vectors are obtained, the model then computes the cosine similarity between the two vectors. A learning-to-rank layer is jointly trained."]},{"cell_type":"code","metadata":{"id":"da5M7bFhqHOc","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":1000},"executionInfo":{"status":"ok","timestamp":1598312854241,"user_tz":420,"elapsed":62827,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}},"outputId":"730ef4d8-a222-406a-8941-323a5c78da82"},"source":["# Install the DeText package\n","!pip install detext==2.0.8"],"execution_count":10,"outputs":[{"output_type":"stream","text":["Collecting detext==2.0.8\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/f2/83/7d78d7663149ba14cd6558a207be934e9d0dd4a9c6f7f9268f4f7ab36fed/detext-2.0.8.tar.gz (66kB)\n","\r\u001b[K     |█████                           | 10kB 13.6MB/s eta 0:00:01\r\u001b[K     |█████████▉                      | 20kB 1.7MB/s eta 0:00:01\r\u001b[K     |██████████████▉                 | 30kB 2.2MB/s eta 0:00:01\r\u001b[K     |███████████████████▊            | 40kB 2.5MB/s eta 0:00:01\r\u001b[K     |████████████████████████▊       | 51kB 2.0MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▋  | 61kB 2.2MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 71kB 2.0MB/s \n","\u001b[?25hRequirement already satisfied: numpy<1.17 in /usr/local/lib/python3.6/dist-packages (from detext==2.0.8) (1.16.6)\n","Requirement already satisfied: smart-arg==0.0.5 in /usr/local/lib/python3.6/dist-packages (from detext==2.0.8) (0.0.5)\n","Collecting tensorflow==1.14.0\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/de/f0/96fb2e0412ae9692dbf400e5b04432885f677ad6241c088ccc5fe7724d69/tensorflow-1.14.0-cp36-cp36m-manylinux1_x86_64.whl (109.2MB)\n","\u001b[K     |████████████████████████████████| 109.2MB 54kB/s \n","\u001b[?25hCollecting tensorflow_ranking==0.1.4\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/be/28/89769cb08c3c04952d3706072ce13b423e09d04d4bd83a7c18fe02ff432a/tensorflow_ranking-0.1.4-py2.py3-none-any.whl (40kB)\n","\u001b[K     |████████████████████████████████| 40kB 4.8MB/s \n","\u001b[?25hCollecting gast==0.2.2\n","  Downloading https://files.pythonhosted.org/packages/4e/35/11749bf99b2d4e3cceb4d55ca22590b0d7c2c62b9de38ac4a4a7f4687421/gast-0.2.2.tar.gz\n","Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0->detext==2.0.8) (1.31.0)\n","Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0->detext==2.0.8) (0.34.2)\n","Collecting tensorflow-estimator<1.15.0rc0,>=1.14.0rc0\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/3c/d5/21860a5b11caf0678fbc8319341b0ae21a07156911132e0e71bffed0510d/tensorflow_estimator-1.14.0-py2.py3-none-any.whl (488kB)\n","\u001b[K     |████████████████████████████████| 491kB 37.6MB/s \n","\u001b[?25hRequirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0->detext==2.0.8) (0.2.0)\n","Collecting keras-applications>=1.0.6\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/71/e3/19762fdfc62877ae9102edf6342d71b28fbfd9dea3d2f96a882ce099b03f/Keras_Applications-1.0.8-py3-none-any.whl (50kB)\n","\u001b[K     |████████████████████████████████| 51kB 6.0MB/s \n","\u001b[?25hRequirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0->detext==2.0.8) (0.9.0)\n","Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0->detext==2.0.8) (3.12.4)\n","Collecting tensorboard<1.15.0,>=1.14.0\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/91/2d/2ed263449a078cd9c8a9ba50ebd50123adf1f8cfbea1492f9084169b89d9/tensorboard-1.14.0-py3-none-any.whl (3.1MB)\n","\u001b[K     |████████████████████████████████| 3.2MB 36.9MB/s \n","\u001b[?25hRequirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0->detext==2.0.8) (1.12.1)\n","Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0->detext==2.0.8) (1.15.0)\n","Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0->detext==2.0.8) (1.1.0)\n","Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0->detext==2.0.8) (0.8.1)\n","Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0->detext==2.0.8) (1.1.2)\n","Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.6->tensorflow==1.14.0->detext==2.0.8) (2.10.0)\n","Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.6.1->tensorflow==1.14.0->detext==2.0.8) (49.2.0)\n","Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0->detext==2.0.8) (1.0.1)\n","Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0->detext==2.0.8) (3.2.2)\n","Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0->detext==2.0.8) (1.7.0)\n","Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0->detext==2.0.8) (3.1.0)\n","Building wheels for collected packages: detext, gast\n","  Building wheel for detext (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for detext: filename=detext-2.0.8-cp36-none-any.whl size=77060 sha256=3f4754954ea5f71bd5d00c1f34d990417e53e4bd5774e484cd4d4668b0d44245\n","  Stored in directory: /root/.cache/pip/wheels/3d/3e/1e/f093368c4b09221180f4e804ff4d766b9f1015f807613c5520\n","  Building wheel for gast (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for gast: filename=gast-0.2.2-cp36-none-any.whl size=7540 sha256=d820a1e58e0558a608c93321ca02d22d8ebd9a34d84af16d06b8ef05abd63b71\n","  Stored in directory: /root/.cache/pip/wheels/5c/2e/7e/a1d4d4fcebe6c381f378ce7743a3ced3699feb89bcfbdadadd\n","Successfully built detext gast\n","\u001b[31mERROR: tensorflow-probability 0.11.0 has requirement gast>=0.3.2, but you'll have gast 0.2.2 which is incompatible.\u001b[0m\n","Installing collected packages: tensorflow-estimator, keras-applications, tensorboard, gast, tensorflow, tensorflow-ranking, detext\n","  Found existing installation: tensorflow-estimator 2.3.0\n","    Uninstalling tensorflow-estimator-2.3.0:\n","      Successfully uninstalled tensorflow-estimator-2.3.0\n","  Found existing installation: tensorboard 2.3.0\n","    Uninstalling tensorboard-2.3.0:\n","      Successfully uninstalled tensorboard-2.3.0\n","  Found existing installation: gast 0.3.3\n","    Uninstalling gast-0.3.3:\n","      Successfully uninstalled gast-0.3.3\n","  Found existing installation: tensorflow 2.3.0\n","    Uninstalling tensorflow-2.3.0:\n","      Successfully uninstalled tensorflow-2.3.0\n","  Found existing installation: detext 2.0.6\n","    Uninstalling detext-2.0.6:\n","      Successfully uninstalled detext-2.0.6\n","Successfully installed detext-2.0.8 gast-0.2.2 keras-applications-1.0.8 tensorboard-1.14.0 tensorflow-1.14.0 tensorflow-estimator-1.14.0 tensorflow-ranking-0.1.4\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"8pOhYvfyqI9A","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":52},"executionInfo":{"status":"ok","timestamp":1598312856018,"user_tz":420,"elapsed":64596,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}},"outputId":"37ca2ef9-6b7c-4d1a-fc2b-5f7a51a3cbbf"},"source":["%tensorflow_version 1.x\n","import detext\n","import tensorflow as tf\n","tf.__version__"],"execution_count":11,"outputs":[{"output_type":"stream","text":["TensorFlow 1.x selected.\n"],"name":"stdout"},{"output_type":"execute_result","data":{"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"},"text/plain":["'1.15.2'"]},"metadata":{"tags":[]},"execution_count":11}]},{"cell_type":"code","metadata":{"id":"hLsUQ9et4o8i","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1598312856019,"user_tz":420,"elapsed":64589,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}}},"source":["# Construct vocabularies from background and training queries\n","vocab_file = 'vocab.txt'\n","vocab = set(['[PAD]', '[UNK]', '[CLS]', '[SEP]'])\n","for query in bkg_queries + train_queries:\n","  vocab.update(query.split(' '))\n","\n","with open(vocab_file, 'w') as fout:\n","  for w in vocab:\n","    fout.write(w + '\\n')"],"execution_count":12,"outputs":[]},{"cell_type":"code","metadata":{"id":"8aviTZbWoKsh","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":68},"executionInfo":{"status":"ok","timestamp":1598313196316,"user_tz":420,"elapsed":404878,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}},"outputId":"b0b808ef-8bff-4397-83df-47df0f787a16"},"source":["def create_bytes_feature(value):\n","  \"\"\"Returns a bytes_list from a string / byte.\"\"\"\n","  return tf.train.Feature(bytes_list=tf.train.BytesList(value=[val.encode('utf8') for val in value]))\n","\n","def create_float_feature(value):\n","  \"\"\"Returns a float_list from a float / double.\"\"\"\n","  return tf.train.Feature(float_list=tf.train.FloatList(value=value))\n","\n","def get_prefix(user_input):\n","  \"\"\"Returns the prefix in the user input. E.g., 'hello w' -> 'hello'.\"\"\"\n","  return ' '.join(user_input.split(' ')[:-1])\n","\n","def get_suffix(text, prefix):\n","  \"\"\"Returns the suffix in the text given prefix. E.g., given text = 'hello world' and prefix = 'hello', suffix_in_words = 'world' \"\"\"\n","  return text[len(prefix):].strip()\n","  \n","max_candidate_num = 5\n","\n","def generate_tfrecord(queries, output_file):\n","  \"\"\"Generates tfrecord given queries\n","\n","  For every query, we split them at every positions to simulate user inputs and generate the completions using the candidate generator\n","  The query before split is the label\n","  For example, given query \"abc\", we will generate user input \"\", \"a\", \"ab\", \"abc\" and the label of the completion \"abc\" will be 1\n","  \"\"\"\n","  writer = tf.python_io.TFRecordWriter(output_file)\n","  total_records = 0\n","  for query in queries:\n","    if len(query) <= 1:\n","      continue\n","\n","    # Split query at every letter to simulate user input\n","    for i in range(len(query)):\n","      user_input = query[:i]\n","\n","      prefix = get_prefix(user_input)\n","\n","      # Generate completions given user input\n","      completions = candidate_generator.search(user_input, max_cost=0, size=max_candidate_num)\n","      \n","      suggested_suffixes = [get_suffix(completion[0], prefix) for completion in completions]\n","      labels = [completion[0] == query for completion in completions]\n","\n","      # Skip cases where no completion is the same as user input\n","      if sum(labels) == 0:\n","        continue\n","\n","      # Compose and write TFRecord\n","      features = {\n","          'usr_prefix': create_bytes_feature([prefix]),\n","          'doc_suffix': create_bytes_feature(suggested_suffixes),\n","          'label': create_float_feature(labels)\n","      }\n","      tf_example = tf.train.Example(features=tf.train.Features(feature=features))\n","      writer.write(tf_example.SerializeToString())\n","      \n","      total_records += 1\n","  print(f'Converted TFRecord to {output_file}. Number of total record: {total_records}')\n","\n","generate_tfrecord(train_queries, './train.tfrecord')\n","generate_tfrecord(dev_queries, './dev.tfrecord')\n","generate_tfrecord(test_queries, './test.tfrecord')"],"execution_count":13,"outputs":[{"output_type":"stream","text":["Converted TFRecord to ./train.tfrecord. Number of total record: 5463\n","Converted TFRecord to ./dev.tfrecord. Number of total record: 2327\n","Converted TFRecord to ./test.tfrecord. Number of total record: 2715\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"gABtO1-QCG2g","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":1000},"executionInfo":{"status":"ok","timestamp":1598313258403,"user_tz":420,"elapsed":466957,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}},"outputId":"970d6ad3-b19e-4a6a-9073-2db675e69070"},"source":["from detext.run_detext import run_detext,DetextArg\n","\n","args = DetextArg(ftr_ext=\"cnn\",\n","        num_filters=50,\n","        num_units=64,\n","        emb_sim_func=[\"inner\"], # cosine matching function\n","        ltr_loss_fn=\"softmax\", # learning-to-rank loss\n","        optimizer=\"bert_adam\", # same AdamWeightDecay optimizer as in BERT training\n","        learning_rate=0.002,\n","        max_len=16,\n","        min_len=3,\n","        use_deep=True,\n","        num_train_steps=300,\n","        steps_per_stats=30,\n","        steps_per_eval=30,\n","        train_batch_size=64,\n","        test_batch_size=64,\n","        pmetric=\"mrr@5\",\n","        vocab_file=vocab_file,\n","        feature_names=[\"label\",\"usr_prefix\",\"doc_suffix\"],\n","        train_file=\"train.tfrecord\",\n","        dev_file=\"dev.tfrecord\",\n","        test_file=\"test.tfrecord\",\n","        out_dir=\"output\")\n","\n","run_detext(args)"],"execution_count":14,"outputs":[{"output_type":"stream","text":["WARNING:tensorflow:\n","The TensorFlow contrib module will not be included in TensorFlow 2.0.\n","For more information, please see:\n","  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n","  * https://github.com/tensorflow/addons\n","  * https://github.com/tensorflow/io (for I/O related ops)\n","If you depend on functionality not listed there, please file an issue.\n","\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/model/lambdarank.py:9: The name tf.layers.Layer is deprecated. Please use tf.compat.v1.layers.Layer instead.\n","\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/optimization.py:164: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["Unable to check if None is of type typing.Union[typing.List[str], NoneType] for field all_metrics of argument class <class 'detext.run_detext.DetextArg'>\n","Unable to check if None is of type typing.Union[typing.List[float], NoneType] for field score_rescale of argument class <class 'detext.run_detext.DetextArg'>\n","Unable to check if None is of type typing.Union[typing.Dict[str, int], NoneType] for field hvd_info of argument class <class 'detext.run_detext.DetextArg'>\n","Unable to check if None is of type typing.Union[typing.List[int], NoneType] for field task_ids of argument class <class 'detext.run_detext.DetextArg'>\n","Unable to check if None is of type typing.Union[typing.List[float], NoneType] for field task_weights of argument class <class 'detext.run_detext.DetextArg'>\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/run_detext.py:213: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/run_detext.py:213: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/run_detext.py:213: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/run_detext.py:213: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/utils/executor_utils.py:16: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/utils/executor_utils.py:16: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Getting executor context info...\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Getting executor context info...\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/run_detext.py:230: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/run_detext.py:230: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/run_detext.py:236: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/run_detext.py:236: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["  saving hparams to output/hparams\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/utils/misc_utils.py:38: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/utils/misc_utils.py:38: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/utils/logger.py:17: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/utils/logger.py:17: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:ParameterServerStrategy with compute_devices = ('/device:CPU:0',), variable_device = '/device:CPU:0'\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:ParameterServerStrategy with compute_devices = ('/device:CPU:0',), variable_device = '/device:CPU:0'\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Initializing RunConfig with distribution strategies.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Initializing RunConfig with distribution strategies.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Not using Distribute Coordinator.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Not using Distribute Coordinator.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Using config: {'_model_dir': 'output', '_tf_random_seed': None, '_save_summary_steps': 30, '_save_checkpoints_steps': 30, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n","graph_options {\n","  rewrite_options {\n","    meta_optimizer_iterations: ONE\n","  }\n","}\n",", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 30, '_train_distribute': <tensorflow.contrib.distribute.python.parameter_server_strategy.ParameterServerStrategy object at 0x7f6b5af386d8>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f6b5af38128>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Using config: {'_model_dir': 'output', '_tf_random_seed': None, '_save_summary_steps': 30, '_save_checkpoints_steps': 30, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n","graph_options {\n","  rewrite_options {\n","    meta_optimizer_iterations: ONE\n","  }\n","}\n",", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 30, '_train_distribute': <tensorflow.contrib.distribute.python.parameter_server_strategy.ParameterServerStrategy object at 0x7f6b5af386d8>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f6b5af38128>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:***** Evaluation on dev set during training *****\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:***** Evaluation on dev set during training *****\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Not using Distribute Coordinator.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Not using Distribute Coordinator.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Running training and evaluation locally (non-distributed).\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Running training and evaluation locally (non-distributed).\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 30 or save_checkpoints_secs None.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 30 or save_checkpoints_secs None.\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/utils/misc_utils.py:332: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/utils/misc_utils.py:332: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/data_fn.py:88: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/data_fn.py:88: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/data_fn.py:79: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/data_fn.py:79: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.sparse_tensor_to_dense is deprecated. Please use tf.sparse.to_dense instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.sparse_tensor_to_dense is deprecated. Please use tf.sparse.to_dense instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/data_fn.py:337: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use `tf.cast` instead.\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/data_fn.py:337: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use `tf.cast` instead.\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n","Instructions for updating:\n","If using Keras pass *_constraint arguments to layers.\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n","Instructions for updating:\n","If using Keras pass *_constraint arguments to layers.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/utils/model_utils.py:19: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/utils/model_utils.py:19: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/model/cnn_model.py:39: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/model/cnn_model.py:39: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/model/cnn_model.py:147: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use `tf.keras.layers.Conv2D` instead.\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/model/cnn_model.py:147: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use `tf.keras.layers.Conv2D` instead.\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Please use `layer.__call__` method instead.\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Please use `layer.__call__` method instead.\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/model/deep_match.py:256: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use keras.layers.Dense instead.\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/model/deep_match.py:256: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use keras.layers.Dense instead.\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/loss.py:43: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use tf.where in 2.0, which has the same broadcast rule as np.where\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/loss.py:43: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use tf.where in 2.0, which has the same broadcast rule as np.where\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/loss.py:47: The name tf.log is deprecated. Please use tf.math.log instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/loss.py:47: The name tf.log is deprecated. Please use tf.math.log instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/train.py:392: The name tf.div_no_nan is deprecated. Please use tf.math.divide_no_nan instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/train.py:392: The name tf.div_no_nan is deprecated. Please use tf.math.divide_no_nan instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/optimization.py:12: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/optimization.py:12: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Getting executor context info...\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Getting executor context info...\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/optimization.py:44: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/optimization.py:44: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/optimization.py:51: The name tf.train.polynomial_decay is deprecated. Please use tf.compat.v1.train.polynomial_decay instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/optimization.py:51: The name tf.train.polynomial_decay is deprecated. Please use tf.compat.v1.train.polynomial_decay instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/train.py:290: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/train.py:290: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/train.py:292: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/train.py:292: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Create CheckpointSaverHook.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Create CheckpointSaverHook.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 0 into output/model.ckpt.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 0 into output/model.ckpt.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.9751147, step = 0\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.9751147, step = 0\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 0, loss = 0.9751147\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 0, loss = 0.9751147\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 10, loss = 0.63269293 (2.571 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 10, loss = 0.63269293 (2.571 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 20, loss = 0.74924386 (0.472 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 20, loss = 0.74924386 (0.472 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 30 into output/model.ckpt.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 30 into output/model.ckpt.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:53:28Z\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:53:28Z\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-30\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-30\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:53:29\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:53:29\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 30: global_step = 30, loss = 0.9049197, metric/mrr@5 = 0.7830611\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 30: global_step = 30, loss = 0.9049197, metric/mrr@5 = 0.7830611\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 30: output/model.ckpt-30\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 30: output/model.ckpt-30\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:## Step 30\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:## Step 30\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.9049196839332581\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.9049196839332581\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-30\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-30\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:keeping checkpoint model.ckpt-30 with metric/mrr@5 = 0.7830610871315002\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:keeping checkpoint model.ckpt-30 with metric/mrr@5 = 0.7830610871315002\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/utils/best_checkpoint_copier.py:73: The name tf.gfile.MkDir is deprecated. Please use tf.io.gfile.mkdir instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/utils/best_checkpoint_copier.py:73: The name tf.gfile.MkDir is deprecated. Please use tf.io.gfile.mkdir instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/utils/best_checkpoint_copier.py:76: The name tf.gfile.Copy is deprecated. Please use tf.io.gfile.copy instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/utils/best_checkpoint_copier.py:76: The name tf.gfile.Copy is deprecated. Please use tf.io.gfile.copy instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/train_helper.py:65: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/train_helper.py:65: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/train_helper.py:71: The name tf.regex_replace is deprecated. Please use tf.strings.regex_replace instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/train/train_helper.py:71: The name tf.regex_replace is deprecated. Please use tf.strings.regex_replace instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Predict: ['prediction', 'serving_default']\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Predict: ['prediction', 'serving_default']\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Train: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Train: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-30\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-30\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Assets added to graph.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Assets added to graph.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:No assets to write.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:No assets to write.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:SavedModel written to: output/export/best_mrr@5/temp-b'1598313209'/saved_model.pb\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:SavedModel written to: output/export/best_mrr@5/temp-b'1598313209'/saved_model.pb\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 3.87267\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 3.87267\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.62327135, step = 30 (7.747 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.62327135, step = 30 (7.747 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 30, loss = 0.62327135 (4.700 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 30, loss = 0.62327135 (4.700 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 40, loss = 0.569968 (0.503 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 40, loss = 0.569968 (0.503 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 50, loss = 0.7470443 (0.506 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 50, loss = 0.7470443 (0.506 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 60 into output/model.ckpt.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 60 into output/model.ckpt.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:53:33Z\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:53:33Z\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-60\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-60\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:53:35\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:53:35\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 60: global_step = 60, loss = 0.86598927, metric/mrr@5 = 0.7909756\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 60: global_step = 60, loss = 0.86598927, metric/mrr@5 = 0.7909756\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 60: output/model.ckpt-60\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 60: output/model.ckpt-60\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:## Step 60\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:## Step 60\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.8659892678260803\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.8659892678260803\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-60\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-60\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:keeping checkpoint model.ckpt-60 with metric/mrr@5 = 0.7909755706787109\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:keeping checkpoint model.ckpt-60 with metric/mrr@5 = 0.7909755706787109\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:removing old checkpoint model.ckpt-30 with metric/mrr@5 = 0.7830610871315002\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:removing old checkpoint model.ckpt-30 with metric/mrr@5 = 0.7830610871315002\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/utils/best_checkpoint_copier.py:121: The name tf.gfile.Remove is deprecated. Please use tf.io.gfile.remove instead.\n","\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/detext/utils/best_checkpoint_copier.py:121: The name tf.gfile.Remove is deprecated. Please use tf.io.gfile.remove instead.\n","\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Predict: ['prediction', 'serving_default']\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Predict: ['prediction', 'serving_default']\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Train: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Train: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-60\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-60\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Assets added to graph.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Assets added to graph.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:No assets to write.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:No assets to write.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:SavedModel written to: output/export/best_mrr@5/temp-b'1598313215'/saved_model.pb\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:SavedModel written to: output/export/best_mrr@5/temp-b'1598313215'/saved_model.pb\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 5.67752\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 5.67752\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.7475408, step = 60 (5.284 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.7475408, step = 60 (5.284 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 60, loss = 0.7475408 (4.273 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 60, loss = 0.7475408 (4.273 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 70, loss = 0.62083983 (0.470 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 70, loss = 0.62083983 (0.470 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 80, loss = 0.7549802 (0.455 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 80, loss = 0.7549802 (0.455 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 90 into output/model.ckpt.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 90 into output/model.ckpt.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:53:39Z\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:53:39Z\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-90\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-90\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:53:40\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:53:40\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 90: global_step = 90, loss = 0.8348177, metric/mrr@5 = 0.80578\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 90: global_step = 90, loss = 0.8348177, metric/mrr@5 = 0.80578\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 90: output/model.ckpt-90\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 90: output/model.ckpt-90\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:## Step 90\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:## Step 90\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.8348177075386047\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.8348177075386047\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-90\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-90\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:keeping checkpoint model.ckpt-90 with metric/mrr@5 = 0.8057799935340881\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:keeping checkpoint model.ckpt-90 with metric/mrr@5 = 0.8057799935340881\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:removing old checkpoint model.ckpt-60 with metric/mrr@5 = 0.7909755706787109\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:removing old checkpoint model.ckpt-60 with metric/mrr@5 = 0.7909755706787109\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Predict: ['prediction', 'serving_default']\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Predict: ['prediction', 'serving_default']\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Train: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Train: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-90\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-90\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Assets added to graph.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Assets added to graph.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:No assets to write.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:No assets to write.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:SavedModel written to: output/export/best_mrr@5/temp-b'1598313220'/saved_model.pb\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:SavedModel written to: output/export/best_mrr@5/temp-b'1598313220'/saved_model.pb\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 5.35902\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 5.35902\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.4066223, step = 90 (5.600 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.4066223, step = 90 (5.600 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 90, loss = 0.4066223 (4.676 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 90, loss = 0.4066223 (4.676 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 101, loss = 0.49790192 (0.476 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 101, loss = 0.49790192 (0.476 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 111, loss = 0.6299217 (0.481 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 111, loss = 0.6299217 (0.481 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 120 into output/model.ckpt.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 120 into output/model.ckpt.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:53:45Z\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:53:45Z\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-120\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-120\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:53:46\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:53:46\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 120: global_step = 120, loss = 0.80456537, metric/mrr@5 = 0.8161223\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 120: global_step = 120, loss = 0.80456537, metric/mrr@5 = 0.8161223\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 120: output/model.ckpt-120\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 120: output/model.ckpt-120\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:## Step 120\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:## Step 120\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.8045653700828552\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.8045653700828552\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-120\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-120\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:keeping checkpoint model.ckpt-120 with metric/mrr@5 = 0.81612229347229\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:keeping checkpoint model.ckpt-120 with metric/mrr@5 = 0.81612229347229\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:removing old checkpoint model.ckpt-90 with metric/mrr@5 = 0.8057799935340881\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:removing old checkpoint model.ckpt-90 with metric/mrr@5 = 0.8057799935340881\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Predict: ['prediction', 'serving_default']\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Predict: ['prediction', 'serving_default']\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Train: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Train: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-120\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-120\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Assets added to graph.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Assets added to graph.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:No assets to write.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:No assets to write.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:SavedModel written to: output/export/best_mrr@5/temp-b'1598313226'/saved_model.pb\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:SavedModel written to: output/export/best_mrr@5/temp-b'1598313226'/saved_model.pb\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 5.3236\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 5.3236\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.30424795, step = 121 (5.632 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.30424795, step = 121 (5.632 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 121, loss = 0.30424795 (4.678 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 121, loss = 0.30424795 (4.678 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 130 vs previous value: 130. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 130 vs previous value: 130. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 130, loss = 0.62638736 (0.496 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 130, loss = 0.62638736 (0.496 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 140, loss = 0.58554757 (0.482 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 140, loss = 0.58554757 (0.482 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 150 into output/model.ckpt.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 150 into output/model.ckpt.\n"],"name":"stderr"},{"output_type":"stream","text":["WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/saver.py:963: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use standard file APIs to delete files with this prefix.\n"],"name":"stdout"},{"output_type":"stream","text":["WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/saver.py:963: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use standard file APIs to delete files with this prefix.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:53:50Z\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:53:50Z\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-150\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-150\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:53:51\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:53:51\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 150: global_step = 150, loss = 0.8480891, metric/mrr@5 = 0.79777974\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 150: global_step = 150, loss = 0.8480891, metric/mrr@5 = 0.79777974\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 150: output/model.ckpt-150\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 150: output/model.ckpt-150\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:## Step 150\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:## Step 150\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.8480890989303589\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.8480890989303589\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-150\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-150\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:skipping checkpoint model.ckpt-150 with metric/mrr@5 = 0.7977797389030457\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:skipping checkpoint model.ckpt-150 with metric/mrr@5 = 0.7977797389030457\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 7.52709\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 7.52709\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.78835493, step = 151 (3.985 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.78835493, step = 151 (3.985 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 151, loss = 0.78835493 (3.004 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 151, loss = 0.78835493 (3.004 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 161, loss = 0.619455 (0.483 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 161, loss = 0.619455 (0.483 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 171, loss = 0.6994833 (0.533 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 171, loss = 0.6994833 (0.533 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 180 into output/model.ckpt.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 180 into output/model.ckpt.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:53:54Z\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:53:54Z\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-180\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-180\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:53:55\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:53:55\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 180: global_step = 180, loss = 0.78444403, metric/mrr@5 = 0.81123763\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 180: global_step = 180, loss = 0.78444403, metric/mrr@5 = 0.81123763\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 180: output/model.ckpt-180\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 180: output/model.ckpt-180\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:## Step 180\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:## Step 180\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.7844440340995789\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.7844440340995789\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-180\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-180\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:skipping checkpoint model.ckpt-180 with metric/mrr@5 = 0.811237633228302\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:skipping checkpoint model.ckpt-180 with metric/mrr@5 = 0.811237633228302\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 7.604\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 7.604\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.5529119, step = 180 (3.949 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.5529119, step = 180 (3.949 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 180, loss = 0.5529119 (2.934 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 180, loss = 0.5529119 (2.934 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 191, loss = 0.46169078 (0.477 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 191, loss = 0.46169078 (0.477 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 201, loss = 0.43987375 (0.474 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 201, loss = 0.43987375 (0.474 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 210 into output/model.ckpt.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 210 into output/model.ckpt.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:53:58Z\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:53:58Z\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-210\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-210\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:53:59\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:53:59\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 210: global_step = 210, loss = 0.7727057, metric/mrr@5 = 0.817433\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 210: global_step = 210, loss = 0.7727057, metric/mrr@5 = 0.817433\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 210: output/model.ckpt-210\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 210: output/model.ckpt-210\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:## Step 210\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:## Step 210\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.7727056741714478\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.7727056741714478\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-210\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-210\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:keeping checkpoint model.ckpt-210 with metric/mrr@5 = 0.8174329996109009\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:keeping checkpoint model.ckpt-210 with metric/mrr@5 = 0.8174329996109009\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:removing old checkpoint model.ckpt-120 with metric/mrr@5 = 0.81612229347229\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:removing old checkpoint model.ckpt-120 with metric/mrr@5 = 0.81612229347229\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Predict: ['prediction', 'serving_default']\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Predict: ['prediction', 'serving_default']\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Train: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Train: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-210\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-210\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Assets added to graph.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Assets added to graph.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:No assets to write.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:No assets to write.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:SavedModel written to: output/export/best_mrr@5/temp-b'1598313239'/saved_model.pb\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:SavedModel written to: output/export/best_mrr@5/temp-b'1598313239'/saved_model.pb\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 5.63726\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 5.63726\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.42213055, step = 211 (5.318 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.42213055, step = 211 (5.318 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 211, loss = 0.42213055 (4.366 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 211, loss = 0.42213055 (4.366 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 221, loss = 0.75956047 (0.472 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 221, loss = 0.75956047 (0.472 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 231, loss = 0.5953821 (0.460 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 231, loss = 0.5953821 (0.460 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 240 into output/model.ckpt.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 240 into output/model.ckpt.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:54:03Z\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:54:03Z\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-240\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-240\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:54:05\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:54:05\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 240: global_step = 240, loss = 0.7789638, metric/mrr@5 = 0.82526135\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 240: global_step = 240, loss = 0.7789638, metric/mrr@5 = 0.82526135\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 240: output/model.ckpt-240\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 240: output/model.ckpt-240\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:## Step 240\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:## Step 240\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.7789638042449951\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.7789638042449951\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-240\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-240\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:keeping checkpoint model.ckpt-240 with metric/mrr@5 = 0.8252613544464111\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:keeping checkpoint model.ckpt-240 with metric/mrr@5 = 0.8252613544464111\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:removing old checkpoint model.ckpt-210 with metric/mrr@5 = 0.8174329996109009\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:removing old checkpoint model.ckpt-210 with metric/mrr@5 = 0.8174329996109009\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Predict: ['prediction', 'serving_default']\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Predict: ['prediction', 'serving_default']\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Train: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Train: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-240\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-240\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Assets added to graph.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Assets added to graph.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:No assets to write.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:No assets to write.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:SavedModel written to: output/export/best_mrr@5/temp-b'1598313245'/saved_model.pb\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:SavedModel written to: output/export/best_mrr@5/temp-b'1598313245'/saved_model.pb\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 5.58211\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 5.58211\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.68949175, step = 240 (5.376 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.68949175, step = 240 (5.376 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 240, loss = 0.68949175 (4.442 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 240, loss = 0.68949175 (4.442 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 251, loss = 0.62329745 (0.461 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 251, loss = 0.62329745 (0.461 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 261, loss = 0.39259064 (0.502 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 261, loss = 0.39259064 (0.502 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 270 into output/model.ckpt.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 270 into output/model.ckpt.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:54:08Z\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:54:08Z\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-270\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-270\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:54:10\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:54:10\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 270: global_step = 270, loss = 0.7663293, metric/mrr@5 = 0.82458806\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 270: global_step = 270, loss = 0.7663293, metric/mrr@5 = 0.82458806\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 270: output/model.ckpt-270\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 270: output/model.ckpt-270\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:## Step 270\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:## Step 270\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.766329288482666\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.766329288482666\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-270\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-270\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:skipping checkpoint model.ckpt-270 with metric/mrr@5 = 0.8245880603790283\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:skipping checkpoint model.ckpt-270 with metric/mrr@5 = 0.8245880603790283\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 7.75406\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step/sec: 7.75406\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.33729923, step = 271 (3.871 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.33729923, step = 271 (3.871 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 271, loss = 0.33729923 (2.910 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 271, loss = 0.33729923 (2.910 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 281, loss = 0.54916704 (0.484 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 281, loss = 0.54916704 (0.484 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 290, loss = 0.3983029 (0.459 sec)\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 290, loss = 0.3983029 (0.459 sec)\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 300 into output/model.ckpt.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving checkpoints for 300 into output/model.ckpt.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:54:12Z\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:54:12Z\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-300\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-300\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:54:14\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:54:14\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 300: global_step = 300, loss = 0.76320726, metric/mrr@5 = 0.82545483\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 300: global_step = 300, loss = 0.76320726, metric/mrr@5 = 0.82545483\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 300: output/model.ckpt-300\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 300: output/model.ckpt-300\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:## Step 300\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:## Step 300\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.7632072567939758\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss : 0.7632072567939758\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-300\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Checking checkpoint model.ckpt-300\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:keeping checkpoint model.ckpt-300 with metric/mrr@5 = 0.825454831123352\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:keeping checkpoint model.ckpt-300 with metric/mrr@5 = 0.825454831123352\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:removing old checkpoint model.ckpt-240 with metric/mrr@5 = 0.8252613544464111\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:removing old checkpoint model.ckpt-240 with metric/mrr@5 = 0.8252613544464111\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Predict: ['prediction', 'serving_default']\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Predict: ['prediction', 'serving_default']\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Train: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Train: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-300\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/model.ckpt-300\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Assets added to graph.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Assets added to graph.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:No assets to write.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:No assets to write.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:SavedModel written to: output/export/best_mrr@5/temp-b'1598313254'/saved_model.pb\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:SavedModel written to: output/export/best_mrr@5/temp-b'1598313254'/saved_model.pb\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Loss for final step: 0.44491068.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Loss for final step: 0.44491068.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:***** Training finished. *****\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:***** Training finished. *****\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Getting executor context info...\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Getting executor context info...\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Initializing RunConfig with distribution strategies.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Initializing RunConfig with distribution strategies.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Not using Distribute Coordinator.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Not using Distribute Coordinator.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Using config: {'_model_dir': 'output/best_mrr@5', '_tf_random_seed': None, '_save_summary_steps': 30, '_save_checkpoints_steps': 30, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n","graph_options {\n","  rewrite_options {\n","    meta_optimizer_iterations: ONE\n","  }\n","}\n",", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 30, '_train_distribute': <tensorflow.contrib.distribute.python.parameter_server_strategy.ParameterServerStrategy object at 0x7f6b5ae88cc0>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f6b5a623860>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Using config: {'_model_dir': 'output/best_mrr@5', '_tf_random_seed': None, '_save_summary_steps': 30, '_save_checkpoints_steps': 30, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n","graph_options {\n","  rewrite_options {\n","    meta_optimizer_iterations: ONE\n","  }\n","}\n",", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 30, '_train_distribute': <tensorflow.contrib.distribute.python.parameter_server_strategy.ParameterServerStrategy object at 0x7f6b5ae88cc0>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f6b5a623860>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done calling model_fn.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:54:16Z\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Starting evaluation at 2020-08-24T23:54:16Z\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Graph was finalized.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/best_mrr@5/model.ckpt-300\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Restoring parameters from output/best_mrr@5/model.ckpt-300\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Done running local_init_op.\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:54:18\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Finished evaluation at 2020-08-24-23:54:18\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 300: global_step = 300, loss = 0.7857997, metric/mrr@5 = 0.81110495\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving dict for global step 300: global_step = 300, loss = 0.7857997, metric/mrr@5 = 0.81110495\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 300: output/best_mrr@5/model.ckpt-300\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:Saving 'checkpoint_path' summary for global step 300: output/best_mrr@5/model.ckpt-300\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:\n","***** Evaluation on test set with best exported model: *****\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:\n","***** Evaluation on test set with best exported model: *****\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 300\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:global_step = 300\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.7857997\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:loss = 0.7857997\n"],"name":"stderr"},{"output_type":"stream","text":["INFO:tensorflow:metric/mrr@5 = 0.81110495\n"],"name":"stdout"},{"output_type":"stream","text":["INFO:tensorflow:metric/mrr@5 = 0.81110495\n"],"name":"stderr"}]},{"cell_type":"code","metadata":{"id":"72yclC-jiBxj","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":0},"executionInfo":{"status":"ok","timestamp":1598313258532,"user_tz":420,"elapsed":467079,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}},"outputId":"019d8355-d3df-454c-9487-662c78be4421"},"source":["# Training output of DeText\n","!ls output"],"execution_count":15,"outputs":[{"output_type":"stream","text":["INFO:tensorflow:best_mrr@5\t\t\t\t     model.ckpt-210.index\n","checkpoint\t\t\t\t     model.ckpt-210.meta\n","eval\t\t\t\t\t     model.ckpt-240.data-00000-of-00001\n","eval_log.txt\t\t\t\t     model.ckpt-240.index\n","events.out.tfevents.1598313201.adb61a8c6e3a  model.ckpt-240.meta\n","export\t\t\t\t\t     model.ckpt-270.data-00000-of-00001\n","graph.pbtxt\t\t\t\t     model.ckpt-270.index\n","hparams\t\t\t\t\t     model.ckpt-270.meta\n","model.ckpt-180.data-00000-of-00001\t     model.ckpt-300.data-00000-of-00001\n","model.ckpt-180.index\t\t\t     model.ckpt-300.index\n","model.ckpt-180.meta\t\t\t     model.ckpt-300.meta\n","model.ckpt-210.data-00000-of-00001\t     network_structure.txt\n","\n"],"name":"stderr"}]},{"cell_type":"code","metadata":{"id":"7SslLTZkP9oi","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":714},"executionInfo":{"status":"ok","timestamp":1598313261639,"user_tz":420,"elapsed":470179,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}},"outputId":"b7a3543a-c45f-4cff-c733-5a246ca0e7eb"},"source":["# Check model signature\n","import glob\n","from tensorflow.contrib import predictor \n","saved_model_path = glob.glob('output/export/best_mrr@5/*')[0]\n","\n","print(saved_model_path)\n","!saved_model_cli show --dir output/export/best_mrr@5/* --tag_set serve --signature_def serving_default"],"execution_count":16,"outputs":[{"output_type":"stream","text":["INFO:tensorflow:output/export/best_mrr@5/1598313254\n","INFO:tensorflow:The given SavedModel SignatureDef contains the following input(s):\n","  inputs['doc_suffix'] tensor_info:\n","      dtype: DT_STRING\n","      shape: (-1)\n","      name: doc_suffix_placeholder:0\n","  inputs['label'] tensor_info:\n","      dtype: DT_FLOAT\n","      shape: (-1)\n","      name: label_placeholder:0\n","  inputs['uid'] tensor_info:\n","      dtype: DT_INT64\n","      shape: ()\n","      name: uid_placeholder:0\n","  inputs['usr_prefix'] tensor_info:\n","      dtype: DT_STRING\n","      shape: ()\n","      name: usr_prefix_placeholder:0\n","  inputs['weight'] tensor_info:\n","      dtype: DT_FLOAT\n","      shape: ()\n","      name: weight_placeholder:0\n","The given SavedModel SignatureDef contains the following output(s):\n","  outputs['label'] tensor_info:\n","      dtype: DT_FLOAT\n","      shape: (1, -1)\n","      name: ExpandDims_2:0\n","  outputs['scores'] tensor_info:\n","      dtype: DT_FLOAT\n","      shape: (-1, -1)\n","      name: Squeeze:0\n","  outputs['uid'] tensor_info:\n","      dtype: DT_INT64\n","      shape: (1)\n","      name: Const_6:0\n","  outputs['weight'] tensor_info:\n","      dtype: DT_FLOAT\n","      shape: (1)\n","      name: Const_7:0\n","Method name is: tensorflow/serving/predict\n","\n"],"name":"stderr"}]},{"cell_type":"markdown","metadata":{"id":"7uUPp8Ub_N-c","colab_type":"text"},"source":["# A QAC System in Action\n","In this section, we put the candidate generation and ranking components together into a QAC system."]},{"cell_type":"code","metadata":{"id":"k4QeFiId3Z6Q","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":122},"executionInfo":{"status":"ok","timestamp":1598313262240,"user_tz":420,"elapsed":470772,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}},"outputId":"edfe1700-7e21-4797-cb27-e991f9326e58"},"source":["# Example of a QAC system\n","predict_fn = predictor.from_saved_model(saved_model_path)  # Load trained TF model\n","\n","def get_completions(user_input):\n","  \"\"\"Generates completions given user input with scores from neural ranking models\"\"\"  \n","  prefix = get_prefix(user_input)\n","  completions = candidate_generator.search(user_input, max_cost=0, size=max_candidate_num)\n","  \n","  suggested_suffixes = [get_suffix(completion[0], prefix) for completion in completions]\n","\n","  predictions = predict_fn( \n","                  {\"doc_suffix\": suggested_suffixes,\n","                   \"usr_prefix\": prefix,\n","                   # dummy pass-through input for label field -- this is not used in making predictions\n","                   \"label\": [1.0]*len(suggested_suffixes)})\n","  \n","  scores = predictions['scores'][0]\n","\n","  completions_with_scores = [(f\"{prefix} {suffix}\".strip(), score) for (suffix, score) in zip(suggested_suffixes, scores)]\n","  completions_with_scores.sort(key=lambda x: (x[1], x[0]), reverse=True)\n","  return completions_with_scores\n","\n","user_input = 'fac'\n","print(get_completions(user_input))"],"execution_count":17,"outputs":[{"output_type":"stream","text":["WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/contrib/predictor/saved_model_predictor.py:153: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.\n","INFO:tensorflow:Restoring parameters from output/export/best_mrr@5/1598313254/variables/variables\n","INFO:tensorflow:[('facebook', 2.0855944), ('facebook com', 1.4921477), ('face painting photos at animal kingdom', -0.0014320329), ('face burn', -0.0014320329), ('face', -0.0014320329)]\n"],"name":"stderr"}]},{"cell_type":"code","metadata":{"id":"4KFLl4FQ9ylS","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1598313262241,"user_tz":420,"elapsed":470766,"user":{"displayName":"Sida Wang","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhX6X9tJZ48VekSiIwpiMJXwreir_J00GucSmeC=s64","userId":"05716004630528480504"}}},"source":[""],"execution_count":17,"outputs":[]}]}